Benchmarking tokens per watt: how AI inference energy efficiency is measured
What benchmarks and measurement methodologies exist for quantifying AI inference energy efficiency in tokens per watt (or per joule), from accelerator to data-centre level, and what do they establish?
The literature on measuring AI inference efficiency in tokens-per-watt terms is young (mostly 2023-2026) and fragmented: one consortium standard exists at the system level (MLPerf Power), but the benchmarks that actually report tokens per watt or joules per token are research tools with incompatible measurement boundaries. Measured numbers span orders of magnitude — roughly 3-4 joules per output token for a 65B model on A100s, 0.002 to 2.9 kWh per 1,000 inferences depending on task, a 65x spread across models in commercial data centres, and a proposed 1/W law under which tokens per watt halves each time the context window doubles. No retrieved benchmark measures tokens per watt at the data-centre (facility) level; facility efficiency is still expressed as PUE, so wall-level tokens per watt is derived, not measured. Confidence is moderate: 20 of 65 sources were read in full text and several prominent items were unreachable in-session.
Updated 17 Aug 202665 sources2009–2026Deep19 min read
tokens per watt · energy per token · MLPerf Power · LLM inference energy · power measurement · PUE · carbon per query · benchmarking